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ABSTRACT 

This report describes a microcomputer system which 
collects data from students in classrooms on a daily basis and is 
then used to evaluate concept achievement and attitude changes 
through a time series analysis. Two pilot studies in two junior high 
schools in Ohio are detailed, where eighth grade students' progress 
in an earth science study unit on the Great Lakes was monitored using 
the study management program. Data from the first study were 
incomplete but did permit some modification of thu program before it 
was tried again. Data for the second study show a baseline pattern of 
decreasing performance which has been found in ov ler studies using 
this data collection system. It appears that students begin to resent 
having to respond to questions on information that they have not been 
taught; they tire of trying to answer the questions correctly; and 
their performance falls. However, in the "intervention" phase of data 
collection, when the unit is being used, the positive influence of 
the unit on the students' knowledge is Indicated by a rising curve. 
It also appears that student performance remains high following the 
end of the unit. The text is supplemented with three figures and an 
eight-item bibliography is provided, (rw) 
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PILOT OF fi SYSTEM FOR COLLECT. ,MG DfilLY 
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Several studies have reported on the development of the 
intensive time series design which allows monitoring class 
achievement and attitudes on a daily basis (Maver and Mon^ 1983). 
The system involves the administration to eac student in class of a 
few objective items on each of the characteristics to be measured. 
Total administration time takes less than five minutes and typically 
occurs at the end of the class period. It has been used to monitor 
concept achievement in six studies. The results have been 
consistent indicating the validity of the design by showing nr. 
effect of frequency of testing upon the data (Mayer and Rr.jas/ 
198.-). The discrimination of the des-gn has also been demonstrated: 
that 15 Its ability to yield data th£.c discriminates between the 
Mlyer"''l934r'''''''^ '"^ ' ^^^''^''** 9^'^"-'Ps '^•^ children (Farnsworth and 

Each of the stud-es has yielded information on learning patterns 
consistent with current understandings of learning and sr-me" 
ir.triq-ung new information suggestive of some pont-treatment 
inip^'ovement in understanding. In the Farnsworth and Mayer s=tudy, for 
example, achievement in understanding of plate tectonics by children 
operatinn at the formal cognitive level continued stable r.r 
accelerated several days into the post-treatment phase. This was not 
true '-^ those children at the concrete cognitive level. Kwon and 
Mayer 198.., reported on a method for identifying and describinq what 
they have called the ■'momentum effect". The design has al^o produced 
interpretable data on attitudes. h - 

Thus far, data have been collected through the use of paper and 
pencil instruments. Thi3 has been very unsatisfactory because of the 
time needed to assemble the instruments and to administer them 
students errors in coding their response and combersome methods of 
handling the data. To overcome these obstacles the use of 
microcomputers for collecting data was investigated. This is a 
report of two pilots of the microcomputer system that was developed 
for use in .col lect ing classroom data for this design. 

5?l?i;ADtiQD_9f _Mi£t:ocorngut er_System 

A system was developed which uses several microcomputers fr,r 
presenting the questions and collecting the response data. It 
consists of two computer programs. The first is an authoring disk 
^c.r preparing up to two item pools of 60 items each. The items in 
each pool can be categorized on two dimensions, eg., by taxonr.rnic 
level of Item and by instructional objective. The second disk is a 
study organization progr? It randomly assigns items to students 
for each day of the study, ensuring an equal distribution of items by 
dimension and also ensuring that no student will get the same item 
more than once during the study. The results and a study management 



program are then transferee! by the study organization program to 
disks for each of the computers that will collect the data. The 
study mangernent program will present the question, allow up to a 
minute for response, record the student response, and also record the 
amount of time taken by the student to respond. 

Six to eight Commodore corrputers, each with a disk drive and 
green screen monitor, were used in the two pilots. *^he teacher was 
required to boot the program disk in the morning and back up data at 
the end of t hp day on a single back-up disk. The back-up procec'ure 
automatically resets each disk for the next days questions. The two 
procedures take no mor^e than cO minutes of the teacher's time each 
day. Whc?n a student types in an identification code, the p^^ograrn 
presents the ques^tion(s) assigned the student for that day. 

Upon completion of the study, data is tr^ansfersd to a data file 
on the university's main computer where it is analysed. Pn added 
feature allows the? teacher to summarise the data on each computer at 
the end of each d^y. Teachers, therefore, can monitor the 
performance? of their classes on a daily basis. 

The two pilots were conducted in two different junior high 
schools in the centr^al Ohio reqion. Both were performed with eighth 
qrade ea^^th sc 3 e^^ice c lassie. The studies were to run for AO days, 
with 10 days as a baseline, 30 days for the intervention and 10 days 
as a foJlow-'Up. fin earth rcience unit was designed focusing on the 
geology, limnology and wej^ihe*" of Lake Erie and the Great Lakes. 
This corrprised the intervention. ft pool of 80 multiple choice items 
were select'^^d from those? used in previous studies. In the first 
pilot half of the item?? directly related to information contained in 
the unit whereas the remainder were related to Great Lakes topics but 
not to those contained in the unit. In the second pilot all eighty 
items related to topics taMqht in the unit. P pool of att'itude items 
using the semantic differential format were also selected from those 
used in previous studies. There were four concepts each with 15 
adjective pairs used as distractors. The four concepts were; Lake 
Erie, today's science cla^^s, today I feel, arid answering questions 
using the computer. Each student responded to one multiple choice 
item and one attitude item each day. 

Pt the end of the intervention, all achievement items were 
administered to all students in each of tne pilots to obtain 
information for item analysis. 

Results^_of^Pi lot^#_l 

The first pilot study was conducted with four classes totaling 
80 students, all taught by the same teacher. It was expected that 
the performance on the unit-speri f ic achievement items would improve 
following *-he beginning of the intervention, whereas performance on 
the other multiple-choice items would remain constant. It was also 
expected that attitudes toward Lake Erie would improve as student 
knowledge of the topic increased. Such changes would be consistent 
with other studies reported in the literature. The remaining 
attitude concepts were expected to vary daily in response to a 
variety of conditions. Unfortunately, the study could not be 



completed. Because of end of year scheduling problems, it started 
with only about 30 days left In the school year. Several of these 
days were lost because of school events. Only £6 days of the study 
were completed. In addition, the teixcher moved through the unit much 
more slowly than originally planned. T> -»refore less than one-half of 
the unit had been completed when school ended. 

When data for the two types of multiple choice items were 
plotted, no trends were appare^'it (Figure 1). filso there were no 
trends for either of the four attitude concepts. Only on response 
time was there a significant and consistent trend. Response time 
dropped the study proceeded (Figure 2). 

»»»»»»»»»»»»»»»»#»»»»»»»»^^»»»*»»» 

INSERT FIGURES 1 PND 2 HERE 

The major benefits from this pilot of the system were to debug 
the study management program^ improve the documentation used by the 
teacher, modify the classroom setup and student identification 
procedures to decrease any problems with student use of the system, 
develop data analysis procedures, test the instructional unit, and 
obtairi item analysis information on the item pools. 

Result s_of_.Pi lot _#£ 

The earth science unit was used with all of the students of an 
earth science teacher. This included six eighth grade classes 
totalling 105 students. Figure three displays the data accumulated 
concerning student ^<nowledge of the uni t . object ives as reflected by 
performance on the multiple-choice items taken every day over the 44 
days of the study. 

**^^***»>**»*-»*'**»»-»^»***»»***»*^^****^^*^^******^*********^****^*^*^* 

INSERT FIGURE THREE HERE 

The first 11 days of the data is the baseline. The data exhibit a 
pattern of decreasing performance. This has been found in each of 
the studies using this data collection system. It appears that 
students begin to resent having to respond to questions on 
information that they have not been taught. As a result, they tire 
of trying to answer the questions correctly, and thus their 
performance falls. The positive influence of the unit on student 
knowledge is apparent from thB gradual rise of the curve during the 
next phase of the data collection, the intervention phase (when the 
unit was being used). It is apparent that student knowledge 
increased gradually as the unit progressed. The highest class 
average was on day 36 (73.7 percent correct), the day that the unit 
posttest was given following the end of the unit. The mean percent 
correct or. the unit posttest was 68.4 percent <50. 4 average raw score 
out of 74 items; standard deviation of 9.98). It also appears that 
the student performance remained hirh following the end of the unit. 
This is the same pattern of performance as reported with previous 
studies using this design with the paper and pencil methods. 
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In previous studies the intensive time-series design had shown 
promise for monitoring concept growth and changes in attitudes- 
However the cumbersome nature of the data collecting procedures made 
it impractical for most clas?=room studies. The two pilot studies 
using the microcomputer ba?i;ed system have demonstrated its 
feasiblity. Using the Commodores makes it relatively cost 
effective. The total investment for equipment was about $6000. The 
fi>""st pilot demon^st rat ed that studrnts soon became accistomed to 
re?:pond3nc] t«-i the items as they were presented by the computer. Thi<=i 
is indicated by the rapid deo-^ease of response time. . The first pilot 
S(l^o revealed bups both in the software and hardware permitting them 
to be worl-ed out. No such prtjblems occurred during the second pilot, 
although one computer did cec"^se to function and was replaced by a 
bacU-np placed a+ the school for that purpose. Results from the 
second p3lot indicate that the data generated by the microcomputer 
systpm 1^ comparable to that [3 pner at ed by the earlier system, 
Therpfo^^p it ^should be equally valid. 

U'^ing the microcomputer system, an intensive time-series study 
can bp developed and administered in a short period of time a^^ruming 
there G\i^ts a ?=uitahle set of multiple choice items. Usin-^ such a 
sysi ern it ir conceivable th^^t researchers and teachers may soon be 
give-n acre«^^ to information, on a daily basis, whtich will indicate 
thc-^ growth of concppt understanding and the fluctuationr of attitudes 
wilhin ? rla^s- For the first time it will be possible to asser:.^ the 
impact of teaching behavior, general classroom environment, and 
student ch .^ract t?r 1 <^t 1 r-s of learning on a day to day basis. 

Nt->w that an efficient and easy to use data collection system has 
been de/is,c?d researchers can design studiecs to examine the impact of 
a varjetv of teaching environments upon learning, n'"'t merely at 
sinyle poi.t^^ before and after intervening but during the ' 
intervention, and on a constant basis, over whatever time interval 
seems appropriate to the study. Future research with this system 
will examine* the types of variables that can be used and ways in 
which they can be measured. One of the next studies will focus on 
adapt ir.n it to collecting word association data permitting its use 
with concept mapping studies. 
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Trend of student knowledge during 
the Evaluation of the GEAGLS Unit. 
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